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ABSTRACT " 

Th^is^^^r^ort concerns the effects of prequestion and 
pbstqu'estion foraats in prose learning in computer-assisted 
instruction. Five experimental groups studied a set of eight passages 
under different prequestion-postquestion combinations. Twenty-five 
subjects, volunteers with high school degrees, were randooly assigned 
to each group* A final retentio^>,test had 32 items classified along 
two dimensions: one dependant on whether the inserted text questions 
prompted attention to the item, and the other dependant on whether 
the item required factual or infer ential ^iearning. The CAI format 
facilitated control over exposure to questicns and passages^ as well 
as data collection on performance measures like referral to passages^ 
time spent on inserted questions, total study time, and scores on 
postquestion sets. A combination of prequestions and postquestions 
led to a 20X increase in student study time relative to the use of 
either question format alone, but did not facilitate relevant 
retention beyond the facilitation found with just prequestions or 
just postquestions. The use of prequestions interfered with 
incidental learning. While equal to the prequestions format in study 
time and effect on relevant learning, the postquestions format seemed 
preferable since it was not accompanied by depressed incidental 
learning* (Author) 
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facilitated control over exposure to question^ aad pas'Sages^, as well as 
data collection on performance measures like referral to passages, time 
spent on inserted questions, total study time, and scores , on postquestion , 
sets. A combination of prequestions and pos tquestions led to a 20% increase 
in student study time relative to the use of either question format alone, 
but did not facilitate relevant retention beyond the facilitation found 
with just prequestions or just postquestions . The use of prequestions 
interfered with incidental learning. l>/hile equal to the prequestions format 
in study time and effect on relevant learning, the postquestions format 
seemed prefereable since it ./as not accompanied by depressed incidental 
learning. 
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The Effects of Pre- and Postquestions or Learning 
from Textual Material in a CAI Format 

INTRODUCTION 

The study described in this report explores the use of questions 
to help students learn from instructional materials presented on a 
computer terminal. In particular, three ways of questioning students 
about the materials they are studying are compared. One questioning 
format presents questions before a student sees the text itself. 
In this format the student may also review these questions while he 
studies the text. A second questioning format displays questions 
only after the student has completed studying the text. In this 
format the student is not allowed to review the text when he uas ^ 
reached 'its questions. The final questioning scheme combines both 
these formats: questions ar^V^splayed before the text and whenever 
the student wishes to review them; after the student completes his 
study of the text questions are presented again. 

Numerous investigations have been concerned with the effects of 
questions available during or after study. The research reported 
here extends these efforts by considering the effects of using questions 
both during and after the study of textual materials. 

After a brief review of relevant research on the use of questions 
as aids to textual learning, the details and results of our 
investigation will be presented. 



REVIEW OF RESEARCH 

Providing students with questions has been shown to enhance 
learning from textual materials (Boyd, 1973; Rickards & Di Vesta, 
1974). Not only is retention of information relevant to the questions 
facilitated by their use, but questions may also facilitate retention 
of passage information that does not answer inserted questions 
(Bruning, 1968; Rothkopf & Udsbiscos, 1967). Rothkopf (1966, 
1972) has attributed the effects of questions to the control which 
they exert over the learner's inspection and processing of the materia 
The position of question^, relative to the text is critical to the 
type of control which is exerted. 
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Questions read before related text segments direct the student's 
attention to specific portions of the text (Frase, 1970), namely 
those portions from which answers to the questions can be derived. 
As^might be expected, readers using such prequestions generally 
demons^5a4:fi--g«ater learning of question-relevant information than 
do those studying without access to questions. But this positive 
increase is fre^quently accompanied by a relative decrease in the 
learning of- information outside the scope of the prequestions. 
Anderson and Biddle (1975) reviewed ,18 studies^ on the use of 
prequestions and incidental learning; they found a depressed level 
of incidental learning reported in 13 of the studies. Thus it 
appears that prequestions induce more attention to selected text 
segments' than would normally be given but may also lead to a 
reduction in the attention given to the remaining segments. 

The effects of prequestions, especially on incidental- learning, 
may be a function of the learner's understanding of the role of 
prequestions relative to the goal of instruction (Duchastel & 
Brown, 1974). If the learner believes the goal of instruction is 
to learn information pertaining to the prequestions, his reading 
is apt to focus strongly on relevant information at the expense of 
incidental information; if ,the reader believes the goal of instruction 
is to learn both incidental and relevant information, his reading is 
apt to be less selective. The directions given students concerning 
the purpose of prequestions relative to the purpose of instruction 
have varied considerably across studies. The majority of investigators 
have associated the two only weakly, if at all. In view of this 
variation" it is surprising that so much evidence confirms the con- 
elusion that prequestions depress incidental learning. As might 
be expected, however, the effect has often been small. In those 
13 studies found by Anderson and Biddle (1975) in which incidental 
learning was depressed, the mean % decrement in scores was only 2.8. 



\ 



- 3 



Questions which follow textual materials effect the learner's 
reading behavior differently than do questions which precede materials. 
Since postquestions are seen af Uer the textual materials have been 

• \ completed, the materials are not studied with the selective attention 
/ found with prequestions. Unlike prequestions, postquestions typically 

/ facilitate questioiv-relevant retention withouL any accompanying 
depression in question- incidental retantion (Rothkopf, 1966). 
Indeed some researchers have found that students given postquestions 
retain, more incidental information than do those not givei' post- 
questions (Rickards & Di Vesta, 1974; Rothkopf & Bisbiscos, 1967) 

The effects of postquestions on question relevant information 
can be attributed to the reader's recall and review of information 
retained in memory that pertains to the questions (Anderson and 
Biddle, 1975). In fact, the recall and review resulting from post- 
questions may aXso involve information incidental but topically related 
to the question-relevant information; McGaw and Grotelueschen (1972) . 
and Rothkopf and Billington (1974) found that postquestions enhanced 
both question-relevant information and information only topically related 
to the relevant information. 

Interest in research exploring the effects of "postquestions 
on question-relevant information has been overshadowed by research 
exploring the effects of postquestions on incidental inf onratioil. 
The latter research has been stimulated^by a number of reports 
that readers receiving postquestions show superior incidental 
learning relative to read-only control students (e.g., Bruning, 1968; 
Rothkopf, 1966; Rothkopf & Bisbiscos, 1967). 

These indirect effects of postquestions have been shown 
to be dependent on both the type of learning required to answer 
the question and the frequency with which/c^uesticns occur within 
the text. For example, Felker and Dapra (11-975) explored the 
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possibility th^t answering ver .atim rather than comprehension post- 
questions could differentially affect students' ability to apply 
, -inciples and concepts discussed in the text. The verb'itim post- , 
questions required recognition of exact text wording while the 
comprehension questions required understanding of the semantic meaning 
of text wording. Both question types were^matched in content and 
focus. On the problem solving test which followed study of the text 
materials, the group using comprehension postquestions outperformed 
the group using verbatim questions. In fact the group with verbatim 
questions performed no better than the read-only control group. 

Rickards and Di Vesta (1974) varied both the ^ type of ' learning 
required to answer the questions and the frequency with v^hich 
postquestions appeared. These investigators used three types of 
postquestions: one type required literal verbatim recall of facts, 
another required literal verbatim recall of ideas, while the third 
required recall and organization of facts supporting ideas. Students 
encountered a postquestion either after every second paragraph of 
text, or after every fourth paragraph of text. When questions appeared 
more frequently, both literal verbatim recall of ideas and recall 
and organization of facts supporting ideas were found to result in, 
significantly more learning of .incidental ideas than literal verbatim 
recall of facts. Furthermore; recall and organization of facts 
supporting ideas resulted in significantly more learning of incidental 
f^cts than the other two activities. But, when questions were asked 
less frequently, incidental retention was the same in all groups. 

One explanation for the indirect effects of postquestions 
is given in terms of shaping (Rothkopf, 1972). According to 
this explanation, a student's reading behaviors preceding those 
post;questions which he can answer are reinforced and maintained, 
-while those preceding questions which cannot be answered are not 



reinforced and become altered. ^-Hiat may result then is that the, 
reader responding to postquestions will attend more than he normally 
would to that type of information to which postquestions are likely 
to refer. Hence, if the incidental 'items on the criterion t^est 
are representative of the postquestions, the .reader who received^ 
postquestions is more likely to have retained the information 
they require. 

With this explanation, a parallel emerges between the effeots 
of pre- and postquestions: where prequestiohs may' focus the cv 
reader's attention on those text stimuli. which provide answers 
to the prequestions, postquestions may focus his attention on 
classes of text stimuli which are likely targets .for future 
postquestions. It can be argue' though that the readerjs 
intention is related to the former effect, while it may not be 
related to- the latter (Rothkopf, 1972)*; whereas the student using 
prequestions chooses to focus his atlen^Eion, the- student using 
postquestions may not be aware of his selective attention. 

Other explanations for the indirect affects associated with 
postquestions have been advanced, including the idea that post- 
questions provide practice for taking the criterion test. 
Indeed; the results of Felker and Dapra (1975) indicate that when 
the demands of the postquestions parallel those of the criterion 
test, incidental learning, will be facilitated. 

In their review of the effects of questions on reading, 
Anderson and Biddle (1975) conclude that postquestions produce 
greater learning than do prequestions. As might be inferred 
from what has been said above, postquestions have almost consistently 

o 

been shown to result in better performance on incidental learning 
than do prequestions. But also- on relevant learning where both 



question fonrats have been shown to have positive results, 
sthere^s a trend for the effects of postquestions to be stronger 
and more consist 3nt than those of -prequestions* 

The findi^^hat postquestions are superior to preqtiestions 
in relevant retention is^ unej:pected when one compares the memory 
requirements involved in answering the questions • As Rickards (1976) 
has noted, in the typical study investigating the effects of pre- 
and postquestions, • the Use of prequestions involves .considerably 
lesstareliance on memory than does the use of postquestions* Students 
are typically not allowed to review the text after chey have reached 
its postquestions. To correctly answer a postquestion then, a reader 
JLs required to remember one or perhaps several items of information 
frota a text ontaiiiing many* other items of inforoiation which have 
competed for his atteni:ion. To Answer a prequestion the reader 
has the text av his disposal; furthermore,* his attention, can focus 
on that information relevant to the question. The option of r viewing 
• the prequestions while reading the tex:t has varied from study to 
study, but even if the reader is not given the option of looking ^ 
back at the prequestions, the memory tequirement imposed may be 
considerably less than is called for in the postquestion format. 
Hence, it would seem' that the probability of learning question- 
relevant information is higher ^when prequestions are used* than when 
postquestions are used. 

That postquestioning is in fact superior to prequestion^ng 
in relevant retention suggests that the study behaviors' resulting 
from postquestions result in more effective learning of the question- 
relevant information than do those, resulting from prequestions. 
\^ereas the student may more often successfully formulate an, answer 
. to a prequestion than a postquestion, whatsis learned by those 
using postquestions is retained more effectively. Apparently, the 
advantage prequestions have* in terms of memory requirements does not 
compensate for the relatively less effective study behaviors which 
they induce. 
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A natural extension to the investigation of pre- and post- 
questioning is the exploration of the effects on studying when 

> 

the two are combined in one treatment. With respect to the 
retention of relevant information, does the study behavior 
invoked by the combination prove sup^ri^ to those behaviors 
associated with either format alone? Ana what will be the effects 
of a combined format on the retention of incidental information? 

Strongly supported answer ,,L^se questions are not available 
as little research has been conducted which explores the combined 
„ questioning format. Boyd (1973) reports one of the few studies which 
compared the combination with either format alone. Based on Boyd's 
results and what Is known about the effects of post quest inning and 
prequestioning, some expectations about the combined formatXwill 
be advanced. 

Boyd found that those given both pre- and postq». stions performed 
significantly better on the retention of question-relevant information 
than did those using only prequestions. The addition of postquestions 
to prequestions adds to the reading behavior typically resulting 
from prequestions further processing of question-relevant information. 
It is expected that this additional processing will result in less 
forgetting of that relevant information. Hence, with respect to 
the retention of relevant infomation, the study behavior invoked by 
the combination should prove superior to that which results 
from prequestions alon^. 

Boyd -also found that those given both pre- and postquestions 
performed significantly better on the reuention of question-relevant 
information than did those using only postquestions. As has been 
mentioned before, the reading behaviors induced by prequestions may 
result in less effective learning of question-relevant information 
than do those which are induced by postquestions. Indeed, Boyd's 
data does indicate that those groups given only postquestions did 
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tend to perform better on relevant retention than did those groups 
given only pfequestions. Apparently the addition of postquestioning 
to prequestioning may compensate for this relatively less effective 
learning and result in a questioning format that is superior to 
postquestioning alone. 

In Boyd's study incidental learning was depressed to the same 
degree in both the groups receiving only prequestions and chose 
receiving both pre- and postquestions. As we have mentioned before, 
prequestions do tend t focus the student's attention on question- 
relevant information. Apparently the addition of postquestions to 
prequestions does little to alter this focusing. The effects of 
a combined format on incidental learning will be similar to those 
of prequestioning alons. 

The primary purpose of out study was to provide more information 
concerning the effects of inserting questions both before and 
after the text. The 'Questions used by Boyd were one-word completions 
requiring verbatim recall. The questions used in our study were 
multiple choice typically requiring a higher level of reader compre- 
hension than is required by verbatim recall. Whether Boyd's re^sults 
would be replicated with different types of questio - was a central 
concern of our investigation. 

A secondary objective of our study was to pursue this 

investigation within the context of computer-as sisted instruction 

and thus extend the range of materials to which results might 
apply. In computer- assisted insti.iction the presentation of text 
is a common instructional event. It was hoped that this research 
would provide authors of computer -assisted instruction programs with 
guidelines concerning optimuio questioning formats to use when textual 
materials are to be presented. 



METHOD 

Experiment Participants . One hundred and twenty-five undergraduates 
from a local four year college participated In the study. The students 
were paid volunteers solicited through classroom announcements and 
advertisements In the campus paper. 

Each student studied a sequence of eight unrelated passages, 
using one of five different question formats. After studying the 
passages, students were requested to take a 32 Item multiple choice 
retention test, with four questions relating to each passage. 

Materials. The eight passages were selected from those used in the 
reading comprehension subtest of the Scholastic Aptitude Test (SAT) . 
The passages chosen had been published in brochures describing the 
test and were no longer being used on forms of the SAT. They each 
averaged about 400 words in length. The topics covered by the 
passages were diverse, with four passages discussing topics in the 
physical and biological sciences, and four discussing topics in the 
social sciences and humanities. This range in topics was intended 
to facilitate Implications for prose learning in general rather than 
for a specific subject matter. 

Questions from the SAT associated with the passages provided 
about 60% of the questions required in the study. The remaining 
questions were constructed by the investigators in consultation with 
professional examiners whose full--time responsibility was test 
development. All questions were multiple choice in format. 

The questions required five types of textual processing: 

(1) comprehension of a supporting idea, explicitly 
stated in the passage, 

^ (2) comprehension of a major idea only indirectly 
expressed in the passage. 
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CS) completion of an inference made in the passage, 

(4) application of a generalization found in the passage 
to a particular instance, 

and (5) evaluation of the logic of the author's discussion. 

Questions of the first type were classified as lower order items. 
Questions of the remaining four types were labeled higher order items. 
The majority of the higher order items were of types 2 and 3. A 
total of 64 questions were used, four lower and four higher order 
items for each passage. 

A better understanding of the question types can be gained through 
some examples. The following question based on a passage about the 
life-style of birds assesses comprehension of a supporting idea 
explicitly stated in the passage (type 1): 

It is essential that birds have* an efficient 
respiratory system because they 

(A) are generally small in si,2e 

* (B) fly in rarefied atmsophere 

(C) have great muscular development 

^ ' t^- (D) must be mentally alert to insure 

survival ^ 

*('E) lead an active existence 
The passage segment from which the question was derived reads: 

Respiration is efficient and indeed must be to 
sustain the high metabolic activity. 

This particular item paraphrases the idea as explicitly stated. Both 
questions using paraphrased restatements and verbatim statements were 
actually used as lower-order items. 

^ An example of an item assessing comprehension, of a main iEfdea 
(type 2) is taken from a passage which critically reviews Pl^^to's 
Republic . Although the actual purpose of the work is not stated, 
the author provides enough information about its ^^tents ^nd the 
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manner in which it is written to allow the following question to be 
answered: 

It is inferred that Plato's Republic is a work written 
principally for the purpose of 

(A) providing citizens with a guide to the best 
possible life 

(B) changing existing methods of education 

*(C) convincing readers that the rule of the few is 
preferable to the rule of the many 

(D) convincing the populace of biological differences 
among classes ^ 

(E) encouraging people to overthrow existing governments 

In a passage concerning the 'development of anesthetics, the 
following statement is made: 

"Curare and its derivatives are being replaced by a 
number of synthetics that are more specific in 
action and predictable in effect." 

Completion of an inference made in the statement is required by the 
following type 3 question: 

It" can be inferred that a disadvantage of using curare 
derivatives as anesthetic agents is that they 

(A) are dangerously explosive 

• (B) are difficult to' manufacture, 

*(C) can produce unexpected effects 

(D) are not as powerful as newer drugs 

(E) cannot be combined with other drugs 

A passage concerning the evolution and operation of cooperatives 
makes the statement: 

A consumers* cooperative sells its goods at the prevailing 
competitive prices in order to avoid conflict with other 
retailers. 
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Application of the rule suggested in this statement concerning 

how a cooperative's prices are determined is required in the following 

type A question: • ^ '* ' . 

Which of the following actions would probably be 
taken by a cooperative in response to a competitive 
price cut by neighboring stores? 

(A) It would maintain its prices ^nd its dividend 
at their regular levels. ■ 

(B) It would maintain its price? at their regular 
level and increase its dividend/ 

*(C) ^ It would cut its prices to equal those of its 
-competitors • 

(D) It would cut its prices to undersell its 
competitors* * 

" ' - (E) It would dissolve. 

An example of the last i,tem type comes from an historical passage 
concerning Puerto Rican politics. In the passage the author uses the 
term "unstable marriage^." The meaning of the term can be determined ^ 
by comprehending implications of stiitements which surround it. The 
following question assesses whether the reader has understood these 
implications : 

The author^prc^pbly describe^ the Alianza as an "unstable 
marriage" in o»er to indicate that 

(A) it had not been officially sanctioned by the 
government 

(B) one faction was 4eliberately undermining the 
united party 

(C) one group had b^en misinformed about the 
party's platfonji ^ 

(D) there had not been unanimous consent to the 
coalition 

*(E) It included two irreconcilable factions 

Each passage had four higher-order and four lower-order ques- 
tions associated with it. The high'er- and lower-order questions for 
a passage were randomly paired, and the pairs randomly assigned a 
number from 1 to 4, The item pairs were then assigned as inserted 
prequestions, inserted postquesticns, and/or retention test items on 
the basis of the numbers they reciived.' Table 1 summarizes the 
assignment scheme of a passage' s^t of item pairs relative to the 
five format groups, lb 
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Table 1 

Question Assignment Relative 
to. Format Groups 



Format 



Assignments 



Prequestions Postquestions 



Retention Items 



CONTROL none 

PRE 1,3 

POST none 

PRE/P6ST(R) 1,3 

PHE/POST (P) 1,3 



none 
none 
1,3 
1,3 

3,4 



1,2 
1,2 
1,2 
1,2 
1,2 



Note . Numbers vefer to item pair assignments. 

The following remarks can all be inferred from Table 1; they 
concern the questions seen by the various groups on a given passage. 
Question pairs 'iT and 2 were used to measure retention for all five 
groups. For all groups except CONTROL, question pair 1 measured 
relevant learning, and question pair 2 measured incidental learning. 

r 

Question pairs 1 and 3 were seen by all groups except CONTROt': as 
prequ.estions by group PRE, as postquestions by group POST, and as 
both pre- and postquestions by group PRE/POST(R). The group 
PRE/POST (P) saw question pairs 1 and 3 as prequestior*?^ but only 
pair 3 was repeated in the postquestioning; they saw the final 
question pair, 4, for their rer/aining two postquestions. Thu3, 
none of this group's postquestions were repeated on the retention test. 

Those stems of questions used as both pre- and postquestions wa?:e 
revised so that the item could be stated as an open-ended- question 
requiring the Student to formulate his own response. This was the 
form in which prequestions were presented, while postquestions were 
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presented in multiple choice format. For example, of the questions 
given as illustrations of the types of questions used in the study, 
4 were selected for use as both prequestions and postquestions. As 
postquestions ^they appeared in the multiple choice format presented 
'above; as prequestions they appeared as follows: 

According to'^ the passage, why is it essential that birds 
have an efficient respiratory system? 

What disadvantage of using curare derivatives as 
anesthetic agents may be inferred from the passage? 

It may be inferred from the passage that a cooperative's 
response to a competitive price cut by neighboring 
stores would be .«.? 

What is the author probably implying with the use of 
"unstable marriage" to describe the Alianza? 

The passages were placed into the topical groupings of physical/ 
biological ,sciences and social sciences/humanities. Item analysis data 
from previous test administrations was available for those passage > 
items borrowed from the SAT. The data was used to order the passages-, 
in each topical set in - terms of the difficulty of their associated 
questions, the passage with easiest questions coming first. This ' 
method ^of .passage ordering was chosen because it was felt that presenting 
progressively more challenging questions was typical of instructional 
materials. Two passage presentation orders were used in the study; 
*both orders maintained the topical grouping orders. One presentation 
order placed all the social science /humanities passages first, followed 
by the physical/biological science passages, while the second isltemated 
between the two topical groups, with the easiest social science/humanities 
passage first*.' 

All passages and inserted question^ were presented to the students 
via a PLATO computer terminal. The retention test was in paper and 
pencil format. The computer presented the study materials as follows: 
all prequestions for a passage were contained on one display, separate 
from the passage; the passages^ were contained on one or two displays, 
depending on their length; each postquestion appeared on a separate 
, display. 

Through the use of selected keys on the terminal keyboard, 
students controlled the rate at which materials were presented. 
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Those having access to prequestions were presented the display of 

a passage's prequestions prior to the presentation of the passage. 

While they studied the passage, those using prequestions were 

free to review them as often as desired. Students were not required 

-tc overtly respond to prequestions. Those having access tc postquestions 

were not allowed to review the passage after they had requested 

the display of its postquestions. Those using postquestions were 

required to answer the postquestions by pressing the terminal key 

designating their alternative choice (a,b,c,d, or e) . Students were 

not allowed to review. a postquestion once they had seen it and had 

gone on to another display. ' - \ 

Question Formats . Five question formats were^ explored . They will be 
denoted CONTROL, PRE, POST, PRE/POST(R) , and PRE/POST(P) . 

CONTROL: Those using the CONTROL format saw no inserted 
questions during their study of the passages. 

PRE: Students studying the passages using the PRE 
format' received prequestions before each passage. 
Four prequestions preceded each passage, two higher 
order and two lower order. The prequestions were 
open-ended requiring the -student to formulate his own 
response. 

POST: Students studying the passage using the POST fonoat 
received postquestions after each ^passage. Four questions 
followed each passage. These were the same questions that, " 
those using the PRE format saw before each passage, but: 
were stated as multiple choice rather than open-ended 
questions. 

PRe7pOST(R) : Students studying the passage using this 
format received both the -prequestions received by the 
'^^RE students and the postquestions received- by the POST 
stuH^ts. The 'R' in the above name denotes the fact ' 
that th^postquestions' repeated the prequestions, i.e., 
there was,^ one-to-one correspondence between the open- - 
ended prequestions and the multiple choice postquestions. 

Half of those questions »?een by^^e student groups described abovje 
also appeared on the retention test-^ 2 from each passage, 1 higher 
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order and 1 lower order. Those questions repeated on the retention 
test appeared in the multiple choice form. 

The PRE group dealt only with the open-ended forms of the 
inserted q'uestions' repeated on the retention test. Because of this, 
if we found that the PRE/POST(R) group outperformed the PRE ?roup, 
some of that improvement could be attributed to the former group's 
practTce"'wi'th-criter ion-like items. .To separate out the effects of 
criterion test' practice pf^yvided by the postqueBtions from the effects 
of further relevant informatiori processi*ng-r.esulting fr^om the post- 
jiluestions, in the PRE/POST(R) format, the 'following "format.-.was^^^ 
included in the study. ; » 

PRE/POST(P) ; For each passage, students studying the 
materials using this format received the prequestion set 
us^d by the PRE and PRE/POST(R) groups; but received 
only two of the postquestions' se.en by -the POST and PRE/ ' 
POST(R) groups. The two postquestAons presented were 
those lower order and higher order , items not repeated 
on the retention test. In addition, two more questions' 
were added^ to the each postquestion set, 1 higher order 
and lower order, that were not used by any of thi: pther • , 
groups and not included on the retention test. (This 
accounts "then for the 16 items whose' function was not 
identified above.) ^ ^ 

Thus those in the PRE/POST(P) group did not receive .as" post- 
questions *any of the items which later appeared .on the retention^ 
test. The 'P' in the above name denotes the fact .that the post- 
questions provided only pi ice. 

The PRE/POST(P) group allowed us to, explore, not otily the effects 
of practice, but also the effects on reading behavior of postquestlons 
referring to prequestion- incidental information. It was Expected * 
that prequestions alone would cause students to ^ocus their attention 
on the prequestiotled infomration to such an extent that the learning 
of incidental information would be impaired. If postquestioris 
repeating the prequestions were added (i.e., the PRE/POST(R) format), 
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the focusing would not be altered. But, If the postquestlon set 
included new questions, would the student not only search for pre- 
question answers, but also prepare for the new questions by studying 
more carefxilly other parts of the text? The investigators did expect- 
that those in the PKe/POST(P) group would indeed attend more to 
prequestiou-incidental information that did those in the PRE or 
PI^/POST(R) groups^ 

Procedures , A PLATO terminal was installed on the campus ♦of the 
local college.;. Student participants were signed-up for individual 
appointments of approximately 90 minutes • They were randomly assigned 
to a question format. Twenty-five students ^^ere included in each 
group. 

When. a student arrived for his appointment, he was seated in 
front oT 'the .terminal and asked to read some directions. Tlie directions 
discussed the purpose 'of " the^experiment and the use of the terminal. 
They also described the question format -th^jt the student would be 
using while he studied the passages. The directions-read by the 
PRE and PRE/POSTCP) groups are included in the appendix of thls^^--....^^^ 

report. An attempt l/as~made to~maintain parallel, if not identical, - - 

directions for each group. 

"'The directions for those students not using the CONTROL format 
indicated that,^after their study of the materials, they would be 
given a test, and on that test would be the questions they had seen 
as inserted questions. There was no indication given of new questions ^ 
appearing on the final test. In order to confirm that students did 
not expect new questions to appear on the retention test, a number 
of participants were asked, after they had completed the retention 
test, if the^^had expected^new. questions to. appear on the test. ^ 
All confirmed 'that they had not expected new questions. 

The directions for (CONTROL students specified that they^ should 
learn as much as they could from the passages. They were warned that 
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a test would follow their study, and on that test would be questions 
asking them to recall facts and ideas stated in the passages and make 
inferences based on things that had been said in the passages. 

After a student completed the directions the principal investigator 
showed him. how to use the keys he would |ieed to work through the 
materials. Also, under the guidance of the investigator, the student 
practiced the key presses on a sample passagerwhicb included those 

inserted questions appropriate to the format to which the student 

■» - 

was assigned. 

All students were given as much time as they needed to complete 
the materials. But, once they had left a passage to go on to the next 
.passage, they were not allowed to return to it. To provide some 
motivation for studying the materials, students were told that if 
they answered more than 40% of the retention test' questions correctly, 
they would receive an extra monetary compensation for their participation 
in the study. In reality all students received the same reimbursement. 

RESUT.TS AND DISCUSSION ' ' 

-Table 1 presents group^means and _ standard deviations for the 

retention test subscores derived from the following sets of items: 
the eight higher order items that also appeared as inserted questions 
(higher order relevant leaming-HRL) , the eight lower order items that 
also appeared as^inserted questions (lower order relevant learning-LRL) , 
the eight higher order it&ns-^that did not appear as inserted questions 
(higher order incidental learning-HIL) ,/and the;. eight lower order 
items that did not appear as inserted questions (lower order »in- - . 
cidental learning-LIL) . Also included in the table are the means 
arid standard deviations for total relevant retention (HFL + LRL) , 
total incidental retention (HIL + LIL) , and total overall retention 
(HRL + LRL + HIL + LIL). The data suggest that, with respect to relevant 
learning, the POST group outperformed those groups given prequestions. 
Furthermore, scores on the higher order questions account for most of 
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the relevant learning difference between the POST group and those 
groups who received prequestions . An unanticipated outcome in the 
data was the poor performance of the PRE/POST with practice group 
in incidental learning. ' The expectation that this group would out- 
perform both the PRE group and the PRE/POST with repetition group 
in incidental learning was not su..ported. A more systematic analysis 
of the retention test scores will begin with some orthogonal contrasts. 
• « 

0 Table 1 

Results on Retention Test: Subscores and Total Score 







RELEVANT 


LEARNING 




INCIDENTAL LEARNING 


OVERALL 


XREATMEirr 


HRL(8)* , 


LRL(8) 


T0TAL(16) 


HIL(8) 


LIL(8) 


T0TAL(16) 


TOTA 


L(32) 




M 


SD • 


M 


SD 


M 


SD 


M 


SD 


M 


SD 


M 


SD 


M 


SD 


POST 


6.04 


1.27 


6.32 


.94 


12.36 


1.80 


4.36 


1.66 


5.32 


1.34 


9.68 


2.34 


22.04 


3.66 


PRE 


5.36 


1.85 


6.20 


1.53 


11.56 


3.00 


3.56 


1.83 


5.12 


1.59 


8.68 


2.85 


20.24 


5.32 


PRE/POST(R) 


5.28 


2.17 


6tp4 


1.67 


11.32 


3.35 


3.72 


1.62 


4.44 


1.83 


8.16 


2.97 


19.48 


5.63 


PRE/POST(P) 


5.20 


1.98 


5.76 


1.62 


10.96 


3.03 


3.28 


1.84 


4.68 


1.77 


7.96 


3.14 


18.92 


5.65 


CONTROL 


4.96 


2.01 


5.12 


1^54 


10.08 


3.07 


4.24 


r.83 


5.12 


1.51 


9.36 


2.77 


19.44 


5.12 



Number in parentheses represents the maximum score possible 
Orthogonal Contrasts for Relevant and Incidental Learning 



Through orthogonal contrasts an attempt was made to answer the 
following questions concerning the retention of question-relevant informa- 
tion:^ 

(1) How does the performance of students with access to questions 
compare with that of students without access to questions? 

(2) How does the performance of students given only post- 
questions compare with th^t of students gi^n _ . • - --^ 
prequeations? - ^'"^ 

(3) How doef the performance of students given postquestions 
as well as prequestions, either in the form of practice 
or repetition, compare with the performance of those 
given only prequestions? 
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(4) When both pre- and postquestions are used, how do 
students given postquestions providing repetition" 
, */ differ in perfprmance from those given postquestions 

• providing practice? ' 

The contrasts underlying these questions reflec^t some of our initial 
expectations* about relative group performance in relevant retention. 
The f^irst contrast allows us to determine whether access to questdQns 
facilitated the retention of question-relevant information. Based , 
on the findings of other investigators, we expected that questioning 
would prove helpful. As was discussed 'in the review of research, 
postquestioning has typically been su^eripr^^/to -prequestioning in, 
facilitating relevant retention; by ' colnparlBg the retention of those 
who received only postquestioning with* those who ,rec^l\(^prequestioning 
with or without postquestioning, the^second cohtra^J^'^llows^s to 
access whether the superiority of postquestiorflTng is supported by 
our data. The third contrast permits the comparison of the c^'ombined 
formats with prequestioning dlone. Since we expected th^ combined 

formats to add to those behaviors typically resulting from pre- 
questioning further proce^s^g of questiou-releyaut information and/or 
criterion test practice, we did expect those using the combined 
format to outperform those using only prequestions . The last contrast 
allows us to compare the effects of criterion test practice with th^ 
effects of further relevant information processing. _ 

Table 2 presents the t-values for these contrasts. 
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^; Table 2 

Orthogonal Contrasts Relevant Learning 



Contrast 
Huaber 



Groups Compared 





Two tailed 


t-Value 


t-Probability 




(df « 120) 


2.27 


•04 


-1.61 


. .12 


- .59 




- .44 





1 POST, PRE, 'PKE/POST(R),''PRE/POST(P) VS. CONTROL 

2 PRE, PRE/POST(R), PRE/POST(P) V.S. posT 

3 ?r<LVPOST(R), PRE/POST(P) , VS. PRE 

4 ,r ■ »^ PRE/POST (P) VS. PRE/POST (R) 



MS Error 

Within Groups « 8.42 



As can be seen from Talkie 2, the results of contrast 1 indicate ' 
that those given questions outperformed those without access to 
questions: providing students with questions did improve question- 
relevant retention.^ Although the retentiou/£est subscores suggested 
that those using ^nly jpostquestlons learned more than tho.se using 
formats which included prequestions, the results of contrast 2 Indicate 



that- this difference is not statistically significant at thfe standard 
.05 level.*- As indicated by the results of the third contrast, the reten- 
tion of those receiving postquestions along with prequpstions was just . 
about the same as those receiving only prequestions. Furthermore, as 
can be seen in the fourth comparison, there i/as no signi^-^.cant difference 
between the pre/post combinations. Since the third contrast suggests 
that the addition of postquestions/ to prequestions h^s little effect 
on relevant retention, it^ would seem to follow that it makes little 
difference on relevant retention, whether the postquestions provide 
both criterion test practice and repetition or just criterion test 
practice. . 

With respect to. incidental learning, our expectations concerning 
relative group performance are reflected in the following questions: 
(1) How does the performance of students given question 

formats including prequestions compare with that of ^ 
those given formats not inclt\ding prequestions? 



(2) How does the performance of studei\ts given only post- 
questions compare with that of those given no 

^ questions?. 

(3) Of those students given prequestions, how does the 
performance of those who receive postquestions referring 
to information incidental to the prequestions compare 

to that of those who do not? 

(4) How do students*" given prequestions differ in performance 
from those given both prequestions and postquestions, 
where the postquestions refer again to the prequestioned 
information? * * ' ' " 

The contrast inferred in the first.. question allows us 

> 

to assess whether prequestioning resulted in depressed incfdental learning 
in comparison to those formats which did not 'include prequestioning. 
Based on the findings of oth^r investigators, we expected prequestioning 
to focus Study behavior to such an extent that incidental learning would, 
be lowered ♦ . The comparison suggested in. the second question between 
the group gi\*en only postquestions and the read-only control group 
attempts to uncover whether postquestioning had the 'indirect* effect 
of facilititing incidental retention. The. third contrast allows us to 
assess whether, in a combined format, the use of postquestions which 
do not match prequestions facilitates incidental retention relative to 
the use of only prequestions or both prequestions and matching post- 
questions. This question, of course, has previously been answered 
in our observations about the retention subscores of Table I; the 
PRE/POST(P) group did not do. as well as did either the *PRE or PRE/POST(R) 
groups on incidental retention. The last contrast addresses the 
expectation that the effect on incidental learning of inserting questions 
both before and after the text will be similar to the effect of 
prequestions alone. 

Table 3 presents the t-vaiues for these contrasts. 
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Orthogonal Contrasts: Incidental Learning 



Contrast 
Number 



Groups Coaipared 



t-Value 



.1 * PRE, PRE/POST(R), PRE/POST(P) VS. POST,' CONTROL 

2 . tOST 7s. CONTROL 

3 PRE, PRE/POST(R) VS. PRE/POST(P) 

4 PRE vs. PRE/POST (R) 



-2. A3 
.40 

- .66 
.65* 



MS error 
Within Groups 



7.99 



Two-tailed 
t-Probablllty 
(df » 120) 



.02 



The results of the first coatrast confirm the expectation that 
prequestions depress incidental learning.. The. results of tlie second- 
contrast indicate that students receiving only poatquestions did 
about as well as the control students on incidental learning; ther^ is 
no evidence that postquestions had the indirect effect of facilitating 
incidental retention above that associated with simply reading a 
passage. Since the PRE/POST(P) format did not result In facilitating 
incidental learning relative to the PRE or PRE/POST (R) format, there 
is no evidence that the PRE/POST^P') format induces a more generalized 
reading strategy than is produced by prequ^astions alone or prequestions 
with repeated postquestions. The results of the last contrast support 
the contention that the processing of incidental information induced 
by prequestions is little effected by the addition of matching 
postquestions. 

Before proceeding to further analyses of the data, a summary of 
the major results disclosed thus far will be given. With respect- to 
relevant retention, it was found that those given questions-performed 
significantly better than did those not given questions. There was 
no evidence, though, that having access to questions both before and 
after text was superior to either using only prequestions or post- 
questions. In fact, trends in the relevant learning subscores 
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indicated that those given only postquestions outperformed those 

given formats including prequestions. Although this difference was 

not statistically significant it is in agreement with the finding of 

other investigators that the effect of postquestioning on relevant 

learning is stronger than that of prequestioning. With respect to r,, 

incidental retention, it was found th^t prequestioning did depress 

incidental learning. On the other hand, there was no evidence that 

the POST format facilitated incidental learning relative to the 

CONTROL format. 

Prequestions vs. Postquestions 

* More information about the question formats can be gained by 
considering just the groups CONTROL, POST, PRE and PRE/POST(R). 
Doing so allows one to conceive of the study as involving two factors, 
where presence of prequestions and absence ot prequestions constitute 
the levels of one factor and presence of postquestions and absencfe 
of postquestions constitute the levels of the other factor. A two . 
factor analysis of variance using these factors was performed for the 
dependent variables incidental learning and relevant learning. In 
this analysis the main effect of prequestions involves comparing the 
average effect of the PRE/P0ST(R) and PRE treatments with that of 
the POST andj CONTROL treatments, while the main effect of postquestions 
involves comparing the average effect of the PRE/P0ST(R) and POST 
treatments with that of the PRE and ^CONTROL treatments. 

The summary table for the analysis of variance using the 
dependent variable of incidental retention appears in Table 4. 
As can be' seen, the only significant difference is the main effect 
of prequestions, reconfirming that formats, including prequestions 
depress incidental learning relative to the no prequestioning formats. 
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Table 4 . 

Analysis of Variance Sumnr^iry Table: 
Presence of Prequestions x Presence of Postquestions 
Depencient Variable: Incidental Retention 













i 


Source of 
Variation 


Sum of 
Squares 


- df 


Mean 
Squares 


F 


•Significance 


r J 

PREQUESTIONS 


30.25 


1 


30.25 


4.02. 


-.05 


POSTQUESTiONS 


.25 


1 


.25 


.03 




^NTEEACTION 


4.41 


1 


4.41 


.59 




WITHIN CELLS 


721.99 ■ 


96 


7.52 







The summary table for the two-factor analysis of variance . 
conducted for the dependent variable of releve^t retention appears 
as Table 5. 



Table 5 

Analysis of Variance Summary Table 
Presence of Preqifestions x Presence of Postquestions 
Dependent Variable: Relevant Retention - 



Source of 
Variation 


Sum.of 
Squares 


df 


Mean 
Squares 


F 


Significance 


PREQUESTIONS 


1.21 


1 


1.21 


- .15 




POSTQUESTIONS 


26.01 


1 - 


26.01 


3.16 


.08 


INTERACTION 


39.69 


1 


39.69 


4.83 


.03 


WITHIN CELLS 


789.19 , 


96 


8.22 
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As can be seen from Table 5, a significant interaction was 
revealed by the analysis. The appearance of an interaction indicates 
that the two treatment means of one factor behave differently finder, 
different levels of the other factor. Based on the results reported 
previously, this interaction is expected; we have already observed 
that the addition of postquestions to a format which has no prequestioning 
results in facilitating relevant retention, while the addition of 
postquestions to a format which has prec^uestioning has little effect 

on relevant retention, figure 1 graphically illustrates how the - 

effects df the presence or absence of postquestions depend on the 
presence or absence of prequestions . 

Figure 1 



Interaction of Prequestions and Postquestions 
in Relevant Learning 
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Additional insight into the interaction is gained through tests of 
simple main effects. These tests are summarized in Table 6. , 
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Table 6 

Tests of Simple Main Effects 
•for Relevant -Learning 



Source 


as 


Q t 


Mo 




L 




PREQUESTIONS (PRE) 


1.21 


1 


1.21 








PRE vs. CONTROL 


27.38 , 


1 


27.38 


3 


.33 


.10 


POST vs. PRE/POST (R) 


13.52 


1 


13.52 


1 


.85 


.25 


POSTQUESTIONS (POST) 


26.01 


1 


26.01 








POST vs. CONTROL 


64.98 


1 


64.98 


7 


.90 


.01 


PRE/POST(R) vs. PRE 


.72 


1 


.72 




.09 




interaction' 


39.69 


1 


39.69 








W. CELL 789.19. 


96 


8.22 









As 3hcwn in Table 6, the difference between the POST and CONTROL 
groups is significant, while that between the PRE and CONTROL groups 
is not significant at commonly accepted levels. There was no significant 
difference in relevant retention between the PRE and PRE/POST(R) 
.group's* Finally, the prequestion/postquestion combination -was not 
significantly different from postquesticns alone. 

How is the lack of significance in the difference between the 

PRE and CONTROL means consistent with the finding presented earlier 

/ 

that questioning has a significant positive effect on relevant 
.retention? You will recall that the conclusion presented earlier was 
based on a more powerful t-test which compared^ the effect of the 
CONTROL group to the average effect of all questioned groups, 
including the relatively higher scoring POST treatment^. Hence it is 
certainly possible that the two tests might result in differing 
conclusions. While the ANOVA does not establish the superiority 
of the prequestioned groups relative to the CONTROL group, it does 
.show that the POST format was reliably better than no questioning; 
this supports the contention that postquestioning has a stronger 
impact on relevant learning than does p'requestioning . 



- 28 - 

Repeated Measures ANOVA: Quesfcion Format, Type of Learning, Level of Question 

' Still more information about the question formats was obtained 
through a 5 X 2 X 2 analysis of variance. The factors were (a) the 
five question formats, (b) the two types of leartiing (inpidental vs. 
relevant) , and (c) the .two levels of questions (higher order 
v.s. lower order), with ^repeated measures on the last two factors. 
Table 7 contains a summary of this analysis. 



Table 7 

Analysis of Variance; Summary Table 
Question Format x Type of Learning x Level of Question 



Source of 
" Variation 


Sum of 
Squares 


df 


Mean 
Squares 


I 


Significance, 


Question Format 


37.29 


4 


9.32 


1.39 




Subj . W. Groups 


802. A A 


120 


6.69 






Type of Lear^ning 




X 




1 97 46 


001 
. \j \j J- 


Type of Learning x 
Question Format 


25.19 


A' 


6.30 


A. 15 


.005 


Type of Learning x 
Subj . W. Groups 


TOO TO 

lo2 .12 


T 0/> 

120 


X • Dl 






Level of Question 


82. A2 


1 


82.42 


51.60 


.001 


Level of Question x 
• Question Format 


7.65 


A 


1.91 


1.20 




Level of Question x 
Subj^. W. Groups 


191.68 


120 


1.60 






Type of Learning x 
Level of Question 


10.66 


1 


10.66 


6.A9 


.025 


Type of Learning x 
Level of Question 
Question Format 


X 

3.13 


A 


.78 


.48 




Type of Learning x 
Level of Question 
Subj. W. Groups 

1 


X 

196.96 


120 


1.6A 










3''l 









As the table indicates, there was^no reliable difference between 
the treatments in totai retention. Relevant learning was significantly 
greater than incidental learning; performance on the. lower order 
questions was significantly higher than perf ormance on the, higher ,order ' 
'.questions. The other two significant findings were the interactions: ^' 
(1) type of learning x question fbrrtat and (2) type of learning x* . 
level of question. We will further explore both of these interactions. 

An understanding of the first interaction is aid^d by the following 
. plot of the relevant retention and incidental retention subscores for ^ 
. the various question formats. 
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Figuife 2' 

Question Format and Retention; 
Relevant and, Incidental Performance 
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As shown in Figure 2, there'is a notable difference in .relevant . 
and Incidental learning throughout the-f.irst four formatSj'.but not in 
the .fifth, the CONTROL format. This is consistent *ith our expecta- 
tions- since the distinction-between-relevant and incidental leaming- 



does not really apply in the control group."- In fact-, tests of simple v 
main effects ~ indicate^ that-- relevant learning was signiiEicantly 
higher than incidental learning at the .001 level for all groups 
except the CONTROL. Tl\ere was no -significant difference between 
relevant and incidental learning in the control group (p^.25). 

Tests. of simple main effects were also per<'ormed to assess 
whether there were differences among the incidental retention means 
or differences among the. relevant retention means. Neither test was 
' significant at the .05 level, although the test comparing the relevant 
retention means approached significance (p^.08") . 

We turn now to the significant type of learning x level of question 
interaction. Tests of simple main effects revealed that relevant . 
retention was greater th^n incidental retention for both higher and 
lower learning' (p:^. 001) ; tests of simple main effects also indicated 
that lower order learning was greater than higher order learning for 

• both j,ncidental and relevant retention (p:^.005). The interaction ' 
detected was due to the fact that the increment, in performance for ' 

* the relevant higher, order items relative'-to the incidental higher order 
items was greater than .that associated with the lower order items' ■ 
(Figure 3) . One possible 'interpretation of this is that higher order 
learning is aid^d more by the use of questions than is lower order 
learning. But competing with this interpretation are two less^ 
interesting explanations. 
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Figure 3 

Type of Learning and Retention: 
Higher-order and Lower-order Performance 
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If the difference in difficulty between the items assigned to measure 
relevant higher order learning and those assigned to measure- incidental 
higher order learning was greater than the difference in difficulty 
between the relevant aijd incidental lower order item sets, an interaction 
of the type represented in Figure 3 might be expected. As item assign- 
ment was random, there is no a priori reason for believing that this 
was in fact the case. But ^ look at the control group's subscores 
(Table 1) will provide"' more concrete information about the differences 
in difficulty between the item subsets. 

In the control group the relevant items were -in no way distinguished 
from the incidental items; hence, for this group, the overall difficulty 
of items measuring relevant learning shoulu match the overall difficulty 
of the itCTis measuring incidental learning. Table 1 indicates that 
the subscore for those lower order items measuring relevant learning 
was equal to the subscore for those lower order items measuring 
incidetttal learning. But' the corresponding higher order subscores 
were not equal. A t-test was performed to test the hypothesis 
that the higher ordler item subsets were in fact equal in difficulty. 
The hypothesis could not be rejected at the .05 level. Hence differen- 
tial item difficulty does not appear to be a tenable explanation for the 
interaction. 
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I. 

Another possible reason for the observed interaction is the tendency 
of subjects to perform near ceiling on the relevant items. Both 
higher and lower order relevant means are artifically depressed by 
the ceiling effect. Furthermore, since lower order"^ items are easier 
than higher order items, the lower order' ^ean is^more^sensitive^o 
the effect than is th'^ higher order mean. This difference in 
sensitivity may account for all, or at least part, of the difference 
in increments between the levels of questions across types of learnings 

The major results brought to light by- the above analyses of 
variance will now be summarized.' Of the relevant learning means, only 
those, of the POST and CONTROL groups were reliably different . The 
significant difference between the POST 'and CONTROL means was established . 
through a direct comparison of the two treatments while exploring fo^; 
simple main effects (Table 6) . A test comparing all 5 relevant learning 
treatment- means was performed as an offshoot of the Question Format x 
Type of Learning x Level of Question ANOVA. This test indicat«^d that 
^ there were no significant differences between ar.y of the treatment 
means. Since the test comparing all 5 means was less/powerful than 
the test comparing the POST, and CONTROL groups directly, U. is not 
inconsistint that the former test does not reestablish the results of 
the latter. 

In comparison to students not having access to the prequestions, * 
those who did experienced depressed incidental learning. This) finding 
was supported earlier in an orthogonal contrast (Table 3) and again in 
the significant main effect of the Prequestions x Postquestions 
ANOVA presented in Table 4. But the comparison of incidental learning 
means done in conjunction with the Question Format x'Type* of Learning 
X Level of Question ANOVA detected no reliable differences between ^ 
the means. Again the results of the three tests can be* considered 
consistent since the orthogonal contrast and the two factor ANOVA ^ 
provided more powerful comparisons of hose receiving prequestions 
'and those not receiving prequestions. 
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•< The Question Format x Type of Learning x Level of Question 
ANOVA detected no reliable differences in the overall learning means 
of the 5 treatments* Furthermore, this analysis indicated that 
lower order learning was greater than higher' order learning for all 
groups, and 'relevant learning was greater than incidental learning 
for all groups except the CONTROL. A Type of Learning x. Level of 
Question interaction was supported by the analysis. One possible 
explanation of the interaction is that higher prder learning benefited 
more by the insertion of questions than did lower order learning. 

Data collected on tae time students spent studying the materials 

1 ' 
ts summarized in Table 8, " 



Table 8 

Mean Time Spent Inspecting Different Parts of the Materials 



Format 


a 
H 


Pre- 

questions* 


Post- 
' questions 


Passages 


Total 


M 


SD 


M 


SD 


M 


SD 


M 


SD 


CONTROL 


22 










36. A5 


10.72 


36. A5 


10^72 


PRE 


22 


13. A2 


A. OA 






AO. 75 


11.16 ' 


5A.17 


1A.51 


POST 


.21 






19. 3A 


A. 92 


38. A6 


8.25 


57 .80 


11.20 


PRE/POST(R) 


21 


11.22 


3.88 


19.98 


7.15 


37.91 


9.68 


69.11 


16. OA 


^RE/POST(P) 


23 


• 9.3A 


3.63 


18.62 


A. 60 


3A.50 


10. AO 


62. A6 


1A.97 



Mote > Mean scores are in minutes. 

\^umber of subjects within each treatment for which complete 
time records werq available. All means within a treatment aire 
based on the subjects with complete records. 



\nien a computer failure occurred while a student was studying the 
materials, the time history collected to that point was lost. The 
proctor reentered the student into the program at that point in the 
materials where he had been prior to the failure. Hence complete 
records were not available for all students. 
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For each set of means, a one-way analysis of variance was used to 
assess their differences. When a significant difference was detected, 
mean comparisons were performed using the Tukey-B test <Winer, 1972). 

There were no significant differences in the postquestion means 
'of the POST, PilE/POST(R) , and PRE/POST(P) groups, or 'the passage 
means of the five groups. There was a significant difference in the 
time the PRE, PRE/POST(R) and PRE/POST(P) groups spent on the 
' prequestions (p:^.01). At the .05 level, the PRE mean was significantly 
greater than the PRE/POST(P) mean, while the PRE/POST (R) mean was not 
reliably diffei;ent from either of the other two means. There was 
also' a significant difference' in the total time the^ive groups took ^ 
to complete the materials (p^.OOl). At- the .05 level, the CONTROL . 
•mean was significantly smaller than the other means, while the PRE/.POST(R) 
mean was significantly greater than the POST and PRE means. No reliable 
differenc'e^s could be detected between the PRE/1?0ST(R) and PRE/POST(P)' 
total time means, or between the PRE, POST, PR£/POST(P) total -time 

means » ^ ' , - > / / 

I The results concerning the time groups spent studying the different 
parts of the materials allow us to reject one .possible explanation »for 
•Why we did not replicate the finding c: Boyd (1973) that the combination 
of pre- and postquestioning -results in superior relevant retention 
relative to either format alone. As "the time data indicates, those / 
in the PRE/POST(R)" grqup spent about the sam< amount of time on the^ • 
prequestions and passa^s as^did those in the PRE J.oup; they spent \ 
about the same amount of time on the postquestions and passages as , 
did those in the POST group. Furthermore, the PRE/POST(R)- group 
spent significantly more time on the task than did either the PRE 
group ov the POST group. Based on study time alone, one might expect ' ■ 
the relevant learning of the PRE/POST(R) group to have been stronger 
than that of the PRE or POST groui-s. But it was not, and it is to 
speculations about why it was not that we now turn. 



ERIC 



Discussion of Relevant Retention and Inserted Questions 

Since the PRE group did as well as the PRE/POST (R) group on 
relevant retention, we may assume that the additional processing 
pr'^vided by postquestionlng had little effect on relevant learning. 
Why\Boyd was able *:o see an" effect while. we were not. may be due to the 
difference between the studies, in, how* students were allowed to use 
prequestlons. In our study,* those using prequestlons could review 
th^ while they studied the text. In -Boyd's- study students were not 
allowed to review preques.tionS once they began th^ text. Because of 
their free. access .to prequestlons, it is likel^j that/those in our 
'^requestioned groupr were better able to formulate answers to the 
prequestlons while they studied the text than were those in Boyd's 
5jtudy. It may be that the Impact of postquestions >in a combined format 
depends on how well stuucnts have been able to formtlate ariswers to 
prequestlons prior to seeing the postquestions. Free access to 
prequestlons may result in learning the question relevant material 
so well that the additional processing resulting from answering the - 
questions again adds little to learning. - .\* 

^ At thr. onset of our study, We did expect the PRE/POST (R) * format . 
'^to result in superior relevant retention relative .to ^ the ^P RE format. 
But we were less certain about how the PRE/POST (R) format would compart 
to the- POST 'format. TM?< extual processing associated with post- 
questioning" typically results in more rerevapt -learning than does that 
associated with prequestioninfe. Since the processing Induced by the 
PRE/POST(R) forma^ is just an epctension of that- induced by prequestign^, 
it is difficult to, speculate which format 'will result in superior 
relevant retention.^ The addition of postquestions to prequestlons in ^' 
Boyd's study appears'tc have compensated ^r the less ef;Eective 
learning resulting; from- prequestioning: those using th,e combined 
form-it outperformed those^usihg only postquestions. 0/}X results, of ^ 
course, lndic;.tG that' the two formats were not significantly different 
in the relevant learning they prqmoted. Why^we did'.not replicate 
Boyd's finding may ''be due to differences in the stlTnulus materials- 
used in the tvq investigations. 



As was discussed In the review of research, Rickards (1976) 
has observed that the f oit^iulation of answers to inserted questions 
may require less reliance on memory when the inserted questions 
appear as prequestions rather than as postquestions . If the stimulus 
materials create a situation in which this advantage results in the 
prequestion group answering more of the inserted questions correctly, 
those using prequestions may learn more relevant information than do 
those using only postquestions. Assuming those using p'f equestions do 
answer more inserted questions correctly than do those using postquestions, 
if review of the priequestions is restricted, those using a combined 
format will be even more likely to outperform those using only postquestions. 

Boyd's mate'."'- s dit. result in those using combined formats 
n*ore sud^^ess fully answering the inserted questions. Evidence for this 
is provided by comparing scores on inserted postquestions for those 
using 'the -combined 'formats with scores on inserted postquestions for 
those using only postquestions. In Boyd's study, those using both 
pre- and pbstquestior^ scored significantly higher on the inserted 
postquestions than did those using only ^postquestions. 

■ ^ • 

, • In our investigation, how the PRE/POST (R) and POST groups compared 
on the performance of inserted questions is siimmarized in Table 9. 
Listed are the means -for the following item subsets: 

-4. 

(1) higher order postquestions not repeated on' the retention 
* ' test,- 

(2) lower order postqu5»stions not repeated on the retention 

test, 

(3) ^higher order postquestions repeated on the retention 

J test, and 

(4) lower order postr.uestions repeated on t;he retention tast. 

t-test indicated thac there were no significant differences between 

the groups on any of the four Subscore means. Apparently, the advantage 

prequestions may have relative to postquestions in placing less demand 

on studeits' memory did not play a significant role in the stimulus 

2 

materials of this investigation.- 
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Table 9 

POST and PRE/POST (R) Inserted Question Means as 
a Function of Appearance of Inserted Questions on the 
Retention Test, and Retention Means of Repeated Items 



/ 

Iton Subset 


POST 


PRE/POST(R) 


Inserted 
Questions 


Retentipn 


Inserted 
Questions 


Retention 


M 


SD 


M 


SD 


M 


SD 


M 


SD 


Higher— Order Item's 


















Repeated 


5.65 


1.23 


6.17 


1.23 


5.65 


1.87 


5.35 


2.12 


Not Repeated 


4.52 


1.47 






4.48 


1.76 






Lower-Order Items 


















Repeated 


6.17 


1.19 . 


6.39 


- .94 


5.96 


1.69 


6.00 


1.70 


Not Repeated 


5.91 


1.41 






"6v4^ 


1.31 







Note > All data is based on tue 23 subjects in each group for 
which inserted question data were available. 



Speculations concerning the retention of relevant information 
experienced by students receiving only prequestions, only postquestions, 
or the combination of prequestions and postquestions will now be 
summarized. It seems that if students are allowed to review prequestions 
while th^y are studying the materials, the addition of postquestions 
to the treatment will not improve relevant retention. If this review \ 
is not allowed, the further processing of question-relevant information^, 
provided by postquestions may improve relevant retention. The addition 
of ptequestions to a postquestion format will typically not improve 
relevant retention. Materials which place gr.eat demands on student 
memory may provide an exception to this generalization. Such demands 
may be made^ for example, by passages which are laden with factual 
information and inserted questions requiring retrieval of selected 
facts. Rickards (1976) provides a concrete example of such stimulus 
materials. If the materials do make demands on students' memory. 
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The question arises as to whether the relative advantage of prequestions 
is a function of whether postquestions require recall, as in BoydVs study, 
or recognition, as in our investigation. Since we did not compare the two 
postquestion types, we have no data upon which to base an answer to this 
question. 4*1 
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the combined format may be superior to only postquestioning but may 
not be superior to only prequestioning. As we have observed before, 
if students are allowed to review prequestions while they study the 
text, additional postquestioning may add little to relevant retention. 

Textual Processing 

At several points in our discussion we have used the conjecture 
that the relevant information processing resulting from postquestioning 
may differ from that induced by prequestions. Some evidence supporting 
this conjecture is provided by comparing the following scores within 
the POST and PRE/POST(R) groups: scores on those inserted postquestions 
which were also retention test items and scores on those same questions 
when they appeared as retention items (Table 9). 

For the 23 students in the PRE/POST(R) group with complete records 
on responses to postquestions, the mean score on the retention test 
for higher order items used as postquestions was 5.35. For these 
students the mean score on the retention test for those lower order 
items used as postquestions was 6.00. Two-tailed t-tests indicated 
that neither subscore was significantly different from its inserted 
question counterpart, although the difference in the higher order 
subscores approached significance (p .09) . ^ 

For the 23 students in the POST group with complete records on 
responses to postquestions, the mean score on the retention test for 
those higher order items used as inserted questions was 6.17. For 
t'aese%tudents, the mean score on the retention test for those lower 
order items used^as inserted questions was 6.39. While the difference 
in the lower order item -subscores was npt significant, the higher 
order subscores .were significantly different at the .002 level. 

\^ile no significant changes seemed to occur in the PRE/PCST(R) 
group between the two question answering periods, the POST group seemed 
. to make gains in relevant higher order learning. If the two groups • 
had processed the question relevant information in the same-^ay, one 
would expect the changes between the two question answering periods 
to be similar. Since th., are not, some support is provided for 
, the alternative hypothesis. 



ERIC 



- 39 - 

CONCLUSIONS - . 

Some conclusions regarding the 'question formats used in the 
study^ will now be drawn. The combined format of prequestioning and 
postquestioning, with the postquestions repeating the prequestions, 
is, in general, an inefficient way to promote relevant learning. 

'^our study it resulted in an approximately 20% increase in student 
study time relative to the use of only pre- or postquestioning but 
did not result in diproved relevant retention relative to these formats. 

The combined format of prequestioning and postquestioning, 
with the postquestions containing 'tiew items, is ineffectual. The 
appearance of prequestion- incidental postquestions does not seem to 
induce ^more regard for incidental information than results in 
groups using only prequestions or prequestions and matching postquestions. 
Furthermore, as indicated by the group's spending significantly less 
time with the prequestions than did the other prequestioned groups, 
the new postquestions seem to induce. *^less regard for the learning of 
relevant information. 

Time spent studying the materials with the use of only post- 
questions was not significantly different from t.ime spent with the 
use of only prequestions. Also there were no significant differences 
in the relevant, learning promoted by tHe two formats. Yet postquestioning 
may be preferable to-, prequestioning for two reasons: 

(1) Postquestioning typically has a stronger positive impact on 
• higher order relevant learning than does prequestioning 

ard (2) Prequestioning depresses incidental learning relative 

to postquestioning. 

As the analysis of time spent studying the textual materials 
indicated, those using only prequestions, only postquestions, or no 
questions at all did not differ reliably in the time they took to 
study the passages. But those who studied with no questions did take 
considerably less study time than the others when time spent on 
inserted questions was added to passage time. The learning advantages 
resulting from inserted questions must be weighed against the extra 
time involved in their use. 
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ThG increnents we observed in relevant learning" which resulted 
from inserted questions are not large. Relative to the control 
group"^, relevant retention increased about 12% in those groups using 
only preque.stions or postquestions. Yet the use of prequestions or* 
postquestions increased the time taken to complete the materials by 
approximately 50%. Was this a worthwhile investment of students^ time? 

With different textual materials and different types of questions 
th^above figures would undoubtably change, but the above question 
would still be relevant. The instructor who contemplates the use of 
questions should assess how difficult it is for the learner to recognize 
what it is he is suppose ^to retain from the materials and hOw critical 
it would be to instructional continuity if certain things from the* 
passage w6re not"^r^t/ined. Such considerations will help the instructor 
decide whether a, perhaps slight i increase in the probability of 
relevant learning is worth the increased study time. 



The increments observed are typical of those commonly found by 
investigators. In those studies reviewed by Anderson and Biddle (1975) 
which compared postquestioning to no questioning, the mean % increment 
in relevant learning for the postquestioned groups was 13.2. The 
percent increment in the present study for POST was 14.3. In those 
studies reviewed which compared' prequestioning to no prequestioning, 
the mean % increment in relevant learning for the prequestioned groups 
was 10.8. The percent increment in the present study for PRE was S.3. 
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Student Directions 



The study you will be a part of concerns methods to help students in- 
learning from written materials presented on a computer display. The 
particular method we are interested in is the use of questions related 
to the materials. 

You will be asked to read eight passages presented on a computer 
terminal. Before each passage you will be given four questions requesting 
information which is either contained in or can be Inferred from the passage. 
(From now on these questions will be called prequestions. ) The prequestions 
are open-ended. They are designed to direct your learning as you read the 
passage. If after studying the passage you feel reasonably confident about 
your answers, to prequestions, then you^ have a sign' that you have studied 
the passage adequately. 

When you have completed reading all eight passages and working through 
th^ir prequestions. you will be asked to take a final test. The test will 
contain the prequestions you have seen, but now they will be restated in 
multiple-choice forrcat . The test will measure how much you remember from 
the information you were directed. to learn from the prequestions. It will 
also conclude the study. 

You will not be allowed to take any notes while you are reading through 
the materials. 

The passages are about 400 words each. Each concerns different and 
unrelated topical areas. Topics chosen are from the humanities, social 
sciences, physical sciences, and biological sciences. 

(As you read through the rest of the directions, you may wish to 
refer to the Key Bpard Directions summarized on the next page.) 

Passage -prequestions will be displayed on the terminal before you see 
their related passage. After reading and familiarizing yourself with these 
prequestions, you can go on to the passage by pressing the 'NEXT' key on 
the computer terminal keyboard. Thereafter, anytime you wish- to review the 
prequestions again, press 'HELP' and they will be displayed again. A press 
of the 'NEXT' key will return you to the passage. 

Some passages are contained on two pages (two separate computer displays). 
You will always begin on page one. If the passage continues on the next 
page, this will be indicated at the bottom of * the first page. The second 
page of a two-page passage will be displayed when you press the 'NEXT' key. 
If you wish to return to the first page from the second you may do so by 
pr:>ssing the 'BACK' key. 

While you are reading through a passage you may review its prequestions 
(by pressing 'HELP') and switch between its pages (by pressing 'NEXT' or 
'BACK'^) as many times as you wish . ' 



After you "have completed the passage and feel like you can answer the 
prequestions, you may indicate this by pressing the 'f (for f^inished)^, key. 
You cannot go back to review the previous prequestibns or passage after 
you have pressed the ^fV'key , so do not press it before you feel you are 
ready to do so. 

After you press the *f' key, the prequestions for the next' passage will 
be displayed and you will begin the procedure again: When you have complete 
the last passage and press the 'f' .key, you will be asked to take the' final 
test. Remember th^t this test will contain the prequestions you have seen 
with each passag^stated in multiple ch6ice form . The final test will be 
given in paper and pencil format. The proctor will give you a copy of the 
test. ^ ' 

If you have a^y questions please direct them to the, proctor. You can 
take as much time as you need to read through the passages. When you are 
ready to begin the passage readings press the 'NEXT' key. 




Key Board Directions 
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To get from^\PREQUESTIONS to PASSAGE press NEXT 

To get from Pj^SAGE to PREQUESTIONS press HELP 

To get from PAGE 1 of a PASSAGE to PAGE 2 press NEXT 

To f$et from PAGE 2 of a PASSAGE to PAGE 1 press BACK 

To get from PASSAGE, to PREQUESTIONS of NEXT PASSAGE press f 
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Student Directions 

The study you will be a part of concerns methods to- help students in. 
learning from written materials presented on a computer display. The 
particular method we are interested in is the use of questions related to 
, the materials. 

You will be asked to read eight passages presented on a computer terminal.. 
Both before and after each passage you will be given four- questions. T^ese 
questions request information which i's either contained in- or can be inferred 
from the passage". (From now on the questions preceding a passage will be 
called prequestions and those following a passage will be called postquestions . ) 
The prequestions are open-ended. They are .designed-to direct your leamifig 
as you -read the passage. The postquestions, on the other hand, are in • ^ . 
multiple choice format. They are designed to serve as check-points' on how . 
well you are learning the materials. If you f«el reasonably -confident about 
your answers to both pre- and postquestions, then you have a sign that you 
have studied a passage adequately. 

There is a connection between the pre- and postquestions. T^^o of the 
postquestiions are merely restatements of two of the prequestions in multiple 
choice format. The other two postquestions are not the same as the prequestions. 

When you have completed reading all eight passages and working through 
the pre- and postquestions, you will be asked to take a final test. The 
te st will contain both multiple choice postquesti ons you have seen as ^ell 
as multiple choice restatements of the passa g e Npreouestions you did not receive 
as postquestions . It will measure how much you remember from the information 
you were directed to learn (from both the pre- and postquestions) . It ; - , 
will also conclude the study. 



You will not be allowed to take any notes while you are reading through 
the materials. " ■ ' 

The passages are abourt'o'o words each. Each concerns different and 
unrelated topical areas. Topics chosen are from the humanities, social 
sciences, physical sciences, and biological sciences. 

(As you read through the rest of the directions, you may wish to refer 
to the Key Board Directions summarized on the next page.) ■ 

• Passage prequestions will be displayed on the -terminal before you see 
their related passage. After Reading and familiarizing yourself with these 
prequestions, you can go on to the passage by pressing the NEXT key on • 
the computer term-nal keyboard. Thereafter, anytime you wish to review the 
prequestions again, press 'HELP' and they will be displayed again. 4 press 
.of the 'NEXT' key will return you to the passage. 

Some passages are contained on two pages (two separate computet displays). 
You will always begin on page one. If the passage continues, on the next 
page, this will be indicated at the bottom of the first pager. The second 
page of a two-page pdssag^ will be displayed when you^ press the NEXT key. 

5 ; . 



If you wish to return to the first page from the second you may do so by • 
pressing, the 'BACK' key. . * .V 

While you are reading through a passage you may review its prequestions 
(by pressing 'HELP') and switch between pages (by pressing 'NEXT ' or 'BACK') 
as many times as you wish . ' ' ^ 

After you have completed the passage and .feel like you can., answer the 
prequestions, you may indicate this by pressing the 'f (fbr f.inished) key. 
^ou cannot go back to review the prequestions or the passage after you have 
pressed the 'f Hey , so do not press It before you,, feel you are ready ^to do so. 

After you press the 'f key the first postquestlon will be displayed. 
Each postquestlon will^be displayed' one at a time. Please indicate your 
answers to postquestlons by pressing the letter (A,B^C,D or. F) of the answer 
you believe is correct. The answer y&a choose will be^^ltten at th^ bottom 
of the screen. While the question is displayed, if you changfe your mind 
about an answer, just press the 'ERASE' key and then press your new answer. 

When you have completed a postqueation a .press of the 'NEXT' key will 
display the next one. You cannot return to a previously displ ayed postquestlon. 
Also you cannot go on to the next postquestlon without answ ering the one^you ^ 
are on . When you have completed the fourth postquestlon a press of the NEXT 
key will display the next set of prequestions and you will begin the procedure 
again. 

When you press the 'NEXT' key after you have completed the la^t post- 
question of the eighth passage you will be asked to take^ the final test.^ 
Remember that this test will contain both multiple dholce nostQuestions you 
have seen as well as multiple choice restatements of/the passag e prequestions 
* you did not receive as postquestlons. 

The final test will be given in paper and pencil format. The proctor 
will give you a copy of the test. 

If you have any questions please direct them to the proctor. Ypu can 
take as much time as you need to read through the* passages . When you are 
ready to begin the passage readings press the 'NEXT', key of the terminal 
keyboard. 

Key Board Directions 
To get from PPEQUESTIONS to PASSAGE press NEXT 

To get from PASSAGE to PREQUESTIONS press HELP 

To get from PAGE 1 of a PASSAGE to PAGE 2 press NEXT 

To get from PAGE 2 of a PASSAGE to PAGE 1 press BACK ^ 

To get from PASSAGE to POSTQUESTIONS press f 

To get from POSTQUESTION to POSTQUESTION press NEXT 

To get from fourth POSTQUESTION to PREQUESTIONS OF NEXT PASSAGE press NEXT 

When answering POSTQUESTIONS, to change an answer first press ERASE and then 
pres^ the new answer (A,B,C,D or E) g 



